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Executive summary 


The project 


Physically Active Lessons (PAL) involves adapting lesson plans to combine short bursts of physical 
activity with academic content. Based on a US programme, PAL is being developed for use in a UK 
primary school context by a research team at the University of Bristol. The intervention aimed to 
increase physical activity by introducing an additional 90 minutes (minimum) per week into the 
teachers’ existing lessons in sessions of approximately 10-15 minutes twice daily, Monday to Friday; 
and to boost attainment levels through improved cognitive function and the consolidation of children’s 
learning. The project was co-funded by Nike inc. as part of the ‘Designed to Move’ initiative. 


Existing research suggests that incorporating moderate to vigorous activity into classroom lessons 
can improve academic achievement; however, the evidence for this is weak and mixed. How such a 
programme might work to improvement academic outcomes for children is also not well understood. 
The study had two aims: (1) to explore the feasibility of adapting a programme of physical activity and 
implementing it in the classroom setting, and (2) to explore the feasibility of a future, large-scale trial 
to provide a proper, robust test of the impact of the programme in improving attainment. 


What did the pilot find? 


Comment 


Question aiaeliare | 


Mixed 


Is the approach feasible? Both teachers and children embraced the programme 
enthusiastically. The programme required a lot of 
additional work from teachers and consequently 
teacher enthusiasm tended to fade. 

Is the approach ready fora No More work is needed to develop the programme before 


full trial? it is ready to be tested using a full trial 


The PAL approach is feasible, however adaptation and further development is required. All teachers 
and pupils embraced the programme with enthusiasm at the beginning, but as the school year 
progressed, enthusiasm started to wane. The difficulties that arose can be condensed into two main 
themes: 


e Firstly, there was no clear set of instructions or support in relation to how to develop and 
implement the programme. This resulted in inconsistent programme delivery across all five 
schools, and, because the programme was still in an early stage of development, it meant there 
was no clear benchmark against which to measure fidelity. At times teachers were unsure 
whether they were delivering the programme correctly, found it challenging to integrate 
activities into lesson plans (especially in the case of literacy), and unclear as to whether the 
programme was achieving the intended outcomes or linking to the curriculum (or teachers’ 
learning objectives) for their pupils. 

e Secondly, teachers had limited time to create and develop new, appropriate activities and 
lessons, implement the programme in the classroom, and share and upload resources to the 
online portal with other teachers taking part in the study. 


The quantitative part of the study showed no differences in literacy and numeracy attainment between 
the intervention group and control group, however, the sub-group analyses suggested that the 
programme might benefit girls more than boys. These results should be interpreted with caution: the 
study design, missing data, and the non-random sample mean that it is not possible to draw firm 
conclusions about the programme’s impact on academic achievement. A full efficacy trial is required 
to establish this. Due to the low proportion of disadvantaged children in the sample it was not possible 
to explore whether the programme might work better for children from deprived backgrounds. 
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How was the pilot conducted? 


This project was set up as a pilot study that aimed to explore the feasibility of PAL in English primary 
schools. As such it does not qualify for an EEF security rating. 


Within each of six primary schools, Years 4 and 5 were randomly allocated to either intervention or 
control groups (472 pupils in total). All schools were therefore involved in the delivery of the 
programme. The children in the treatment group received the PAL programme for two terms, between 
February and July 2014. The children in the control group did not receive the intervention and 
represented ‘business as usual’. This was a post-test only design and data was collected in July and 
October 2014. One of the six schools dropped out after randomisation and before the programme 
started. Another school withdrew during data collection and so data for their control pupils was not 
collected. This missing data potentially introduces considerable bias into the quantitative results. 


A process evaluation was conducted alongside the trial in order to provide data on programme 
implementation and to capture the perceptions and experiences of key stakeholders’ engagement 
with the programme. Key areas explored were the quality of implementation of the intervention, 
fidelity, and perceived programme outcomes. All headteachers, as well as intervention and control 
teachers in participating schools, were invited to take part. Three headteachers, all seven intervention 
teachers and two control teachers agreed to participate. Three focus groups were conducted with 6- 
10 children in each. 


Recommendations for the development of the programme 


Many of the issues that arose in the teacher and headteacher interviews can be addressed through 
the development of a programme manual alongside the provision of additional mentoring and support. 
Such a manual would provide teachers with a clear understanding of the programme’s aims and 
outcomes as well as a comprehensive guide on how to implement it, including integrating activities 
into numeracy and, especially, literacy lessons. Finally, a stronger mentoring and support structure 
needs to be put in place throughout the delivery phase to ensure fidelity and quality of 
implementation. 


How much does it cost? 


The cost of the programme as delivered in the one-year pilot study was estimated at £9.08 per pupil. 
This estimate is based on 237 pupils receiving the programme, this being the first year of 
implementation, and includes two initial half-day training sessions. These sessions were provided by 
an external trainer, costing £2,150 in total. This estimate does not include direct salary costs, supply 
cover for training, or the out of school time required for teachers to develop the necessary resources. 
Once teachers have been trained to deliver the programme there are no on-going costs to the school. 
On this basis, the estimated cost of implementing the programme over three years works out at £3.02 
per pupil per year. 


Key Conclusions 

1. Evidence for the feasibility of implementing Physically Active Lessons in the classroom is mixed; 
both teachers and children reported enjoying the programme, however, the programme created a lot 
of additional work for teachers and thus, enthusiasm tended to fade towards the end of the study. 

2. Teachers found the training enjoyable and helpful, however, they reported that it was sometimes 
difficult and often time consuming to develop lesson plans and think of ways to integrate physical 
activity into lessons, especially literacy lessons. 

3. More work is needed to develop the programme before it is ready to be tested using a full trial. 
Additional work is required to develop a programme manual and integrate additional support 
mechanisms for teachers throughout the delivery period of the intervention. 

4. If the programme is taken to a full effectiveness trial it is recommended that paper-based tests are 
used, additional explanatory variables are measured, and randomisation can successfully occur at the 
year-group level. 
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Introduction 


This is a report of a pilot study of the Physically Active Lessons programme, which was piloted in five 
primary schools in Hertfordshire between February and July 2014. 


Background and context 


There is evidence to suggest that school-based physical activity programmes can result in small 
improvements in certain health and wellbeing outcomes for children. The evidence relating to the 
impact of such programmes on academic achievement, however, is more limited and generally 
inconclusive. Only a small number of studies have been conducted in the area and the majority of 
these have not employed rigorous designs or explored long-term impacts. 


Schools provide a unique setting for accessing young people and are therefore an ideal setting for 
physical activity interventions (Marmot, 2010). In a recent study, international experts compiled a 
ranked set of 29 physical activity research priorities (Gillis et al., 2013). The top three priorities were: 
(1) the development of effective and sustainable school-based interventions to increase physical 
activity; (2) policy changes to influence physical activity in children; and (3) retrospective, longitudinal 
studies of the independent effects of physical activity and sedentary behaviour on health (Gillis et al., 
2013). 


There has been a decline in the physical activity of primary school children. Currently, only half of all 
seven-year-olds meet the level of physical activity recommended by the Chief Medical Officer’s 
(CMO) physical activity guidelines, with girls less active than boys (Griffiths et al., 2013). A recent 
systematic review of school-based physical activity programmes (including 44 randomised controlled 
trials with over 36,000 participants) concluded that there is some evidence of a positive effect of these 
programmes on pupil health and wellbeing outcomes (Dobbins et al., 2013). These include: 
increasing the duration of physical activity per day, increasing aerobic capacity (as an indicator of 
physical fitness), and decreasing the amount of sedentary behaviour (Dobbins, et al., 2013). 
However, results should be interpreted with caution as the included trials had generally a small 
magnitude of effect and a moderate risk of bias (due to over-reliance on self-report for behavioural 
outcome data, high attrition rates, incomplete outcome data and lack of blinding of outcome 
assessors). Similarly, another systematic review has suggested that integrating physical activity into 
the daily school curriculum has beneficial effects in terms of preventing child obesity, particularly 
among primary school children (Waters et al., 2011). There is further evidence, albeit limited due to a 
lack of high quality studies, that physical activity reduces anxiety and depression in children and 
adolescents as well as producing short-term improvements in self-esteem (Biddle and Asare, 2011). 


The potential benefits of physical activity may extend beyond those established for the health and 
wellbeing outcomes described above, and there is some limited evidence to suggest an association 
between physical activity and cognitive function (Tomporowski et al., 2008; Martin et al., 2014; 
Rasberry et al., 2013; Singh et al., 2012; Lees and Hopkins, 2012; Trudeau and Shephard, 2008). A 
recent systematic review (Martin et al., 2014) identified four randomised controlled trials that 
assessed the impact of school-based weight management interventions (targeting physical activity 
and diet) on general school achievement. Whilst there was no evidence of any improvement in 
general achievement across these studies, one study (N = 96) involving a 13-week programme of 40 
minutes of vigorous physical activity for five days per week (Davis et al., 2011) indicated an increase 
in mathematics achievement, executive function, and working memory among overweight pupils. 
This study was the most methodologically rigorous and, of those studies included in the review, the 
least at risk of bias. The authors noted that, while the effects were small—in the region of 0.2 
standard deviations—large numbers of children and adolescents could benefit from such 
interventions. It is worth pointing out however, that while the authors suggest these effects are small, 
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improvements of this magnitude are generally considered to be educationally meaningful, equating to 
approximately three months’ progress. 


Whilst a small number of RCTs have shown that physical activity can result in improvements in 
cognitive functioning, including working memory and executive function (Such as Martin et al., 2014; 
Fisher et al., 2011), these trials have tended to be small with at least moderate risk of bias. It is 
thought that improvement across these specific domains of cognitive functioning may help explain 
improvements in academic achievement, however, the relationship between physical activity, 
cognitive functioning and academic achievement is undoubtedly complex, not yet well understood and 
requires further, theory-based, exploration (Tomporowski et al., 2011). 


Intervention 


It is with the above in mind that this evaluation seeks to explore the role and feasibility of a classroom 
based programme of physical activity. The PAL programme involves developing and adapting lesson 
plans to combine short bursts of physical activity with academic content. It has the dual aim of: 


e increasing physical activity by introducing an additional 90 minutes (minimum) per week into 
teachers’ existing lessons in sessions of approximately 10-15 minutes twice daily, Monday to 
Friday; and 

e boosting attainment levels through improved cognitive function and the consolidation of 
children’s learning. 


The types of activities and how they are integrated into the curriculum can vary, for example pupils 
might do star jumps while recalling the times tables during a numeracy lesson. More examples of the 
types of activities used by the programme are provided in Table 1 in the methodology section and a 
sample lesson plan can be found in Appendix 3. 


Existing evidence for the intervention 


PAL is based on a US programme called Physical Activity Across the Curriculum (PAAC) that evolved 
from a combination of existing lessons from teachers with examples of activities from Take 10!, a 
program that integrates physical activity, nutrition, and health concepts with lessons in primary school 
classrooms. * 


In a review of Take10! (including variations of it) by Kibbe et al. (2011), three of the ten evaluations 
explored the impact of the programme on academic achievement (Murray et al., 2008; Hollar et al., 
2010; Donnelly et al., 2009) and only one of these studies used a randomised controlled trial design 
(Donnelly et al., 2009) (described below). Using a quasi-experimental design and focusing on children 
at risk (n = 193), Murray et al. (2008) reported improvements in both numeracy and reading, however 
they did not report an effect size or provide sufficient information for one to be calculated. Similarly, 
Hollar et al. (2010) evaluated a variation of the Take 10! Programme—also using a quasi- 
experimental design—which involved five schools over a two-year period. In addition to nutrition 
information, schools implemented physical activity programmes including Take10!. Based on a 
subsample of 350 primary school pupils they reported considerable improvement in numeracy (ES d = 
0.5) and smaller improvements in reading (ES d = 0.19). 


PAAC, as noted above, evolved from Take10!. It was originally developed by teachers and aimed to 
impact on learning and physical activity by incorporating ten minutes of movement into the school 
day. PAAC aims to reduce body mass index, increase physical activity and fitness as well as improve 
academic attainment in primary school children; it involves 90 minutes per week of moderate to 


| Take 10! is a series of multimedia resources and has been used extensively by schools across the UK for a number of years 
to teach physical literacy and activity (further information at www.take10.org.uk). 
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vigorous physically active academic lessons. A three-year cluster randomised controlled trial of PAAC 
(involving 24 US elementary schools in the US) found that in fact body mass index did not reduce 
over three years as a result of taking part in the programme (Donnelly et al., 2009). It did however 
appear to improve academic outcomes—measured from baseline to three years and using the 
Weschler Individual Achievement Test for a subset of the sample (n = 203). This is a smaller sample 
size than the pilot study reported here, and results showed that the children who received PAAC 
performed significantly better than the control children across reading, numeracy and spelling. Only 
change scores are reported in the paper and there is insufficient information to calculate an effect 
size. 


Finally, a quasi-experimental evaluation (n = 8 schools) of a similar intervention to PAL, called Texas 
I-CAN! (Bartholomew and Jowers, 2011), showed that programme implementation improved when 
teachers received more in-depth training in relation to the underlying theory of the programme as well 
as more explicit, specific and appropriate activities that they could more easily integrate into lesson 
plans. Compared to sedentary lessons, physically active lessons were positively related to children’s 
physical activity (as measured by steps taken) in addition to cognitive improvements—sustained 
attention as measured by time on task. Time on task was assessed through a series of five-second 
interval observations of intervention and control children by two trained observers. No information was 
provided relating to the timeframe over which the observations were recorded, or the testing 
conditions. Insufficient information was reported to calculate an effect size estimate. 


Stage of development of the intervention 


As outlined above, Take10! has been shown to improve attainment in numeracy by an effect size of 
up to d = 0.5 and attainment in reading by an effect size of d = 0.2 (Hollar et al., 2010). This is the 
best available estimate, however it should be borne in mind that the study was quasi-experimental in 
design meaning that this estimate is likely to be inflated and biased. Additionally, Take10! also 
included nutrition and health concept lessons and so it is not possible to attribute this effect size solely 
to the physical activity component of the programme. 


To establish whether similar effects might be achieved in a UK-based sample, a research team at the 
University of Bristol adapted PAAC for a UK primary school context. The PAL programme has not 
been implemented in schools in the UK setting prior to this evaluation. The programme is still in 
development and as yet does not have a manual detailing how it should be implemented (in other 
words, it is not a ‘manualised’ intervention). Indeed one important element of the design of the 
delivery was that teachers should play an active role in developing and creating activities and 
resources that could be used by other teachers who were delivering the programme. The programme 
requires further development before it will be ready for an efficacy or effectiveness trial. For these 
reasons the current study was a pilot study that facilitated the development of the programme whilst 
also exploring the feasibility of an experimental test of its efficacy in the future. 


Rationale for conducting the evaluation 


The purpose of the current, independent evaluation was to determine the feasibility of adapting, 
implementing and, in the future, evaluating this intervention using a full scale randomised controlled 
trial design. The specific aim of the study was twofold: 


¢ to explore the feasibility of programme adaptation and implementation through interviews and 
focus groups; and 
¢ to explore the practicality of a future, large-scale trial through a small pilot study. 
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The study took place between October 2013 and October 2014 and comprised both qualitative and 
quantitative data collection. 


Evaluation objectives 
The implementation element of the study aimed to: 


¢ inform and support the development and adaptation of the intervention for a UK context 
through independent observations of programme development, teacher training and 
programme implementation; 

¢ explore the barriers and challenges associated with implementation, and identify potential 
solutions through interviews with programme developers, teachers and headteachers; 

¢ determine whether the intervention was delivered with fidelity through observations of 
programme delivery and interviews with teachers; 

¢ explore whether the intervention can be scaled-up, using interviews with programme 
developers, teachers and headteachers; and 

¢ determine the acceptability to intervention pupils, teachers and headteachers through 
interviews and focus groups. 


The trial element of the study aimed to: 


¢ inform the selection of outcomes and measures through piloting the administration of potential 
educational and affective outcome measures—specifically, this involved comparing the ease 
of use and administration of digital versus paper tests to measure educational outcomes; 

¢ determine the feasibility of the randomisation process—that of allocating year groups within 
schools to the intervention and control groups, through interviews with headteachers and 
teachers; 

* provide initial quantitative outcome data on the efficacy of the programme through the 
analysis of the quantitative data collected on educational and affective outcomes at the end of 
the programme; and 

¢ advise the University of Bristol research team and the EEF on the requirements and process 
for undertaking a full scale randomised controlled trial (RCT). 


Project team 


The project was conceived by the team at the University of Bristol, led by Raj Chande (School of 
Economics, Finance and Management). The University of Bristol team was supported by the Teacher 
Development Trust (TDT) in the recruitment of schools, the training of teachers to deliver the 
programme, and by providing ongoing support for implementation. 


The independent evaluation team was led by Dr Sarah Miller at Queen’s University Belfast, who was 
supported by Aideen Gildea, Dr Seaneen Sloan and Prof Allen Thurston. The evaluation team was 
responsible for the design of the evaluation methods and all data collection for both the impact 
evaluation and the process evaluation. 


Ethical review 


Ethical approval was granted by the Research and Ethics Committee of the School of Education, 
Queen’s University Belfast, in November 2013, prior to any data collection being undertaken. 
Informed consent was obtained at the school level from headteachers and teachers. Written 
information and opt-out consent forms were sent home to parents of eligible pupils (Appendix 2). 
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Methodology 


Trial Design 


The evaluation was designed as a pilot study to determine the potential of incorporating physical 
activity into the teaching of the normal curriculum. The study involved two year groups (Year 4 and 
Year 5) in a convenience sample of six primary schools. The unit of randomization was the year group 
and in each school Years 4 and 5 were randomly allocated to either intervention or control. All schools 
were therefore involved in the delivery of the programme. The children in the treatment group 
received the Physically Active Lessons (PAL) programme for two terms between February and July 
2014. The children in the control group did not receive the intervention and represented ‘business as 
usual’. 


Eligibility 


The TDT recruited primary schools from the Hertfordshire area, an area in which they had existing, 
established links with schools. School recruitment took place between October and November 2013. 
Primary schools were eligible to take part in the evaluation if they: (1) had at least one form entry, (2) 
were willing to be randomly assigned to condition at the year group level, (3) were willing to engage 
with the programme and implement it with Year 4 or Year 5 pupils, and (4) were willing to allow the 
evaluation team access to pupils for administration of tests. 


An information event was held for teachers from schools that had expressed an interest in taking part. 
This was organised by the University of Bristol project team and the TDT, and was also attended by 
members of the evaluation team. This provided an opportunity to explain both the programme and the 
nature of the independent evaluation. 


School-level consent for the implementation of the intervention (from headteachers and participating 
class teachers) was sought prior to randomisation. A Memorandum of Understanding (MoU) that 
detailed schools’ responsibilities and rights regarding the project was signed by all participating 
headteachers (Appendix 1). 


Primary schools informed the parents of all Year 4 and Year 5 pupils about the evaluation using 
information and consent forms provided by the evaluation team (Appendix 2). Parents had the 
opportunity to withdraw their child from the evaluation by returning the opt-out consent form to the 
class teacher. No opt-out parental consent forms were returned. 


Intervention 


PAL is a classroom-based intervention that is delivered by teachers to primary school pupils. It 
integrates short bouts of physical activity with academic content related to literacy and numeracy. The 
aim of PAL is to introduce an additional 90 minutes per week of physical activity into the teachers’ 
existing lessons in sessions of approximately 10—15 minutes twice daily, Monday to Friday. 


In this study, programme implementation and fidelity were supported by two half-day training events 
in addition to an online portal and forum for teachers to share their activities and experiences. The 
first half-day training session, aimed solely at intervention teachers, took place prior to commencing 
the programme in January 2014. It was supported by the TDT and delivered by a specialist trainer, 
Patricia Maude MBE, a physical education consultant from the University of Cambridge who is widely 
published in the areas of physical literacy and children’s movement development. A second half-day 
training event was provided midway through the academic year (March 2014). Some of the Physically 
Active Lessons suggested at the training are illustrated in Table 1. At this halfway point, the 
intervention included only the activities described below and some suggested ways of incorporating 
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these with academic content. The practicality of how the activities would be integrated with literacy 
and numeracy lessons was still to be developed by participating teachers. How this was achieved in 
practice is illustrated through the example lesson plan in Appendix 3. 


Table 1: Example of physical activities that could be incorporated into academic lessons 
Legs and Feet 


Stepping Jumping 
Stepping on the spot Bounce 
Walk Hop on right/left foot 
Marchiwith high knees/with straight leg lift Jump from 2 feet to 2 feet 
Jog Jump 2 feet to 2 feet with turns 
Stepping to sides and with turns Jump 2 feet to L/R 
Stepping forward and back Hop scotch 
Star jumps 
Fronts, Backs and Sides Arms and Hands 
Straight shape, arms stretched up Wave arms above head/up and down 
Star shape Swing arms back and front 
Small shape Clap hands above head 
Twisted shape Snap fingers 
Forward fold Pass beanbag from hand to hand 
Side bend to right/left Send up, clap and catch beanbag 


It is important to note this was not a manualised programme at the outset of the evaluation. One of 
the aims of the study therefore was to explore the feasibility of the delivery of PAL, where teachers 
would use a collaborative enquiry framework, supported by the TDT, to develop their own 
activities/resources as well as adapt those that were suggested at the training. In addition to this, the 
online portal was intended to provide a forum through which teachers could share and disseminate 
activities, and to discuss any problems they might be encountering. The overall aim was that the 
content on the website/portal at the end of the implementation year would help develop PAL activities 
and resources suitable for a UK context for a forthcoming trial. 


Teachers were trained to deliver the programme in January 2014, and, after a two-week development 
and practice period, began to implement the programme with their pupils from February to July 2014. 
Teachers in the control condition were asked not to attempt to implement any elements of the 
programme with their classes during this time. 


Outcomes 


The primary outcomes were attainment in literacy and numeracy, and these were measured at post- 
test only. The feasibility of two reliable’ and standardised methods of administering assessments 
were tested: 


1. Pencil and paper administration, using Progress in English (PiE; short-form) 9 and 10 and 
Progress in Maths (PiM) 9 and 10, both published by GL Assessment (gl-assessment.co.uk). 

2. Computerised administration, using the computer-adaptive InCAS tests, developed and 
distributed by the Centre for Evaluation and Monitoring (CEM) at Durham University 
(cem.org). 


A secondary outcome was enjoyment of school. For pupils who completed InCAS this was measured 
by the attitude subscale of the INCAS assessment (namely, enjoyment of school). Pupils who 
completed the paper tests also completed the Enjoyment of School scale developed by Pell and 


- For PiE and PiM Reported reliability ranges from (Cronbach's alpha) 0.80-0.91. Reliability of INCAS tests reports Rasch 
Person between 0.97 and 0.89. 
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Jarvis (2001) and used in several previous trials in education. This is a reliable scale with reported 
Cronbach’s alpha in the range of 0.65 to 0.73 (Pell and Jarvis, 2001; Miller and Connolly, 2012). 


All tests were completed on a whole class basis under exam conditions. In three of the five schools 
the evaluation team administered and scored the tests. In the remaining two schools the teaching 
staff in the school administered the tests. Three schools completed paper and pencil tests, and two 
completed digital tests. The sixth school withdrew from the study after randomisation and prior to the 
start of the intervention (see below for details). 


It was originally intended that additional health data would be collected by the British Heart 
Foundation at pre- and post-test, and that time on task data would be collected by the University of 
Bristol research team. Due to feasibility and time constraints, this data was not collected and 
consequently is not included in this report. 


Sample size 


Previous evaluations of the PAAC programme have reported effect sizes in the order of d = 0.5 in 
favour of the intervention group in numeracy (see Hollar et al., 2010). This may be an inflated 
estimate due to the design of this and the data on which it is based, it is however the best available 
estimate. 


Within each school, one year group (Year 4 or Year 5) was allocated to the treatment and the other 
year group acted as control. Whilst the main aim of this pilot study was not to determine the 
effectiveness of the intervention, it had 45% power to detect an effect size of d = 0.5 based on the 
following assumptions (calculated using Optimal Design (Version 3.01)): 


¢ Significance level (a) = 0.05 

¢ Number of students per class = 30 

¢ Number of year groups = 12 

¢ Number of schools = 6 

e Effect size (6) = 0.50 

¢ Estimated intraclass correlation coefficient (9) = 0.04 


Randomisation 


The six schools were matched and paired according to school size. Simple randomisation (using the 
sampling function in Stata 12) was then used to allocate schools within each pair to deliver the 
intervention to either their Year 4 or their Year 5 group (with the remaining year group in each school 
acting as a control). The year group, rather than classes within year groups, were chosen as the unit 
of randomisation. The reason for this was twofold: firstly there was potentially a higher risk of spill- 
over effects between classes within the one year group, and secondly, not all schools had two-form 
entry. The evaluation team undertook the allocation process and the TDT, who recruited the schools, 
communicated the results to the schools. 


Analysis 


Analysis was conducted in Stata version 13 (Stata Corporation, College Station, Texas, USA), on an 
intention-to-treat basis. The main effects of the intervention were estimated using a clustered 
sandwich estimator (Williams, 2000) to take account of the hierarchical structure of the data and the 
fact that the observations within clusters cannot be considered to be independent. Failing to account 
for this hierarchical structure within the data risks biased standard error estimates and inflates the 
Type | Error rate. This can result in spurious statistical significance and an erroneous rejection of the 
null hypothesis—in other words, concluding that there are between-group differences, when in fact 
none exist (Donner, 1998). 
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Models were estimated for each outcome measure. For each model, the relevant outcome measure 
at post-test formed the dependent variable and a number of independent variables were added 
including: a dummy variable representing whether the child was a member of the intervention or 
control group (coded ‘1’ and ‘0’ respectively); and a series of other covariates representing the 
students’ core characteristics (gender and FSM eligibility). The main focus for the analysis was the 
estimated coefficient associated with the dummy variable that represented the difference in mean 
scores on the respective outcome variable between the intervention and control groups, once FSM 
and gender were controlled for. This coefficient was then used to estimate the effect size of the 
programme in relation to the respective outcome variable as the standardised mean difference 
between the two groups (Hedges’ g). 


In addition to the analysis of the main effects, an exploratory analysis was planned to examine any 
differential impact of the intervention relating to gender or FSM eligibility. Standardised scores from 
the two separate literacy and numeracy tests were combined into one score for literacy and one for 
numeracy, and it is recognised that analysing data from different tests may impact on the validity of 
the results. 


Process Evaluation Methodology 


A process evaluation was conducted alongside the trial in order to provide data on programme 
implementation and to capture the perceptions and experiences of key stakeholders’ engagement 
with the programme. Key areas explored were the quality of implementation, fidelity, and perceived 
programme outcomes. All headteachers, intervention teachers, and control teachers in the five 
participating schools were invited to take part: three headteachers, all seven intervention teachers, 
and two control teachers agreed to participate. Three focus groups were conducted with 6—10 
children in each, with the aim of ascertaining pupils’ views of the programme. Focus groups were 
undertaken with specific children in the intervention group purposively selected by the teacher to 
represent a range of observed responses to the programme (from resistance or non-response to 
active engagement). Only two of the participating schools agreed to a lesson observation. The 
remaining schools (n = 3) did not participate in a lesson observation because at that stage of the year 
they were either no longer delivering the programme or were only delivering it sporadically and 
delivery did not coincide with the planned research visit. It should be noted that the lesson 
observation was not a structured means of data collection but was for the purpose of enabling the 
researchers to contextualize the programme in action. Data was collected through a variety of 
activities at different time points as shown in the table below. 
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Table 2: Timeline of activities related to the process evaluation 
Activity 


January Teacher training observation 
2014 


March 2014 _ First telephone interview with intervention teachers 


April 2014 Second telephone interview with intervention teachers 


May-July Face-to-face interview with headteachers, intervention and control teachers, and focus 


2014 group interviews with children in the intervention group 
May-July Observation visits of Physically Active lessons. 
2014 


Due to the small number of schools taking part in the study, all participating schools were involved in 
the process evaluation. Process evaluation data was collected and analysed by the evaluation team. 


Protocol violations 


The original protocol stated that prior attainment and ethnicity data would be collected. Due to time 
and cost considerations this element of the proposal was revised: this data was not collected and so 
this analysis was not possible. 
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Impact evaluation 


Timeline 


Table 3: Timeline of activities related to the trial 
Date Activity 


October to November 2013 Recruitment by University of Bristol team and TDT 
November 2013 Randomisation by evaluation team 
“January2014 t—~—t Teacher training 
“February to July2014.——~™*” Programme delivery 
Juy2014 7 Outcome data collection (3 schools) 
“October2014.=—C Outtcome dataccollection completed (2 schools) 
“November 2014. ——s—i‘<‘«‘C*CO ret report to EEF 
“March2015 i (stst—‘—s~™ Expected date of publication 


Participants 


The Teacher Development Trust initially approached nine primary schools in the Hertfordshire area 
between October and December 2013. A letter was sent to each school outlining the purpose of the 
study and the implications of participation. This letter was followed up by a telephone call. Of these 
nine schools, six agreed to take part and each headteacher was asked to sign a Memorandum of 
Understanding (Appendix 1). This number of schools was deemed appropriate given the nature of the 
study—a pilot study of an intervention in the early stages of development. 


Table 4: Characteristics of participating schools 


Total % KS2 KS1- 
No. boys rexey tay KS2 


pupils scores VAS" 
2013 


Year 5 Year 4 
(n=49) (n=57) 
2 Community Year 4 Year 5 221 
(n=29) (n=30) 
3 Foundation Year 4 Year 5 236 
(n=28) (n=28) 
4 Community Year 5 Year 4 451 
(n=59) (n=54) 
5 Community Year 5 Year 4 501 
(n=61) (n=60) 
6 Community Year 4 Year 5 306 
(n=30) (n=30) 


tValue Added Score 


Foundation 434 


Of the six schools that were recruited, four were Community Schools and two were Foundation 
Schools. This is not a random sample of primary schools and therefore is not representative of all 
primary schools in the Hertfordshire area or indeed England. The majority of schools had relatively 
low proportions of children who had special educational needs (SEN), low proportions of children for 
whom English was not their first language (EAL), and a low proportion of children eligible for free 
school meals (FSM) compared to other state primary schools in England. Only one of the six schools 
(School 6) had a high proportion of EAL pupils, and a higher than average proportion of FSM pupils. 
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Data collection and missing data 


Unfortunately, post randomisation, School 6 withdrew its consent to take part in both the intervention 
and data collection due to unforeseen circumstances resulting in unexpected staff changes. In this 
school, the Year 4 pupils had been allocated to the intervention group and the Year 5 pupils to the 
control group. This withdrawal left five schools remaining in the study: two schools delivering the 
intervention to their Year 4 pupils and three schools delivering the intervention to their Year 5 pupils. 
Consequently the balance between the schools delivering the programme to Year 4 and Year 5 was 
disrupted such that three of the remaining schools were delivering the programme to Year 4 and two 
were delivering it to Year 5. 


At post-test data collection, all five remaining schools provided attainment data: 


¢ Two schools completed paper and pencil tests in June/July. 

* One school completed digital tests in June with their Year 4 intervention group and then 
withdrew, providing no data for the Year 5 control group (n = 40). 

« The two remaining schools—also assigned to administer the digital tests—encountered 
difficulties in doing so (see below for more detail) and completed their data collection in 
September/October instead. One of these two schools opted to complete the paper-based 
test rather than the digital test. 


The staggered data collection may well have had implications for the comparability of test scores 
collected in June 2014 (n = 3 schools) and those collected in October 2014 (n = 2 schools). This 
effect might also have been exacerbated by ‘summer learning loss’ as the interval coincided with 
school holidays, possibly resulting in poorer test performance in October compared to June. 


All three of the schools that encountered difficulties in completing the data collection had been 
assigned to complete the digital tests and this was directly related to their inability to complete the 
tests in June or July. This mode of testing was more time consuming, as the number of pupils that 
could be tested at once was limited to the availability of computers or laptops. This often meant that 
pupils had to be tested in groups, which was disruptive for the class and required an additional 
member of staff to oversee testing. There were also technical issues, such as pupils being unable to 
log on to the school network, laptop battery failure and headphones not working. Schools also felt that 
the set-up process was time consuming and required specialist technical input, which was only 
available in the school on a limited basis. The schools that used pencil and paper tests reported no 
difficulties administering tests. 


Pupil characteristics 


Of the five schools remaining in the study, three schools delivered the intervention to their Year 5 
pupils (with Year 4 pupils acting as controls) and two schools delivered the intervention to their Year 4 
pupils (with Year 5 pupils acting as controls). In total, 372 out of a possible 412 pupils from nine year 
groups and five schools completed outcome measures; intervention n = 205, control n = 167. 
Intervention and control groups are broken down by year group and gender in Table 5, below. 
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Table 5: Characteristics of participating pupils in the final (analysed) sample 


Intervention Oxo) alice) | 
NC) NC) 
Year group 
Year 4 68 (33.2%) 141 (84.4%) 
(Mean age 9yrs 6mths) 
Year 5 137 (66.8%) 26 (15.6%) 
(Mean age 10yrs 5mths) 
Total 205 (100%) 167 (100%) 
Gender 
Male 103 (50.2%) 93 (55.7%) 
Female 102 (49.8%) 74 (44.3%) 
| Total 205 (100%) 167 (100%) 
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Figure 1: Flow diagram of participants through the study 


Assessed for eligibility (n=9 
schools) 
Excluded (n=0) 


Not meeting 
inclusion — criteria 
(n=0) 


Declined to 
Randomised (n=6 schools, participate  (n=3 
n=12 year groups, n=472 schools) 
pupils) 


Allocated to intervention (n=6 year Allocated to control (n=6 year groups, 
groups, n=235 pupils) n=237 pupils) 


Received allocated intervention (n=5 Received allocated intervention (n=5 year 
year groups, n=205 pupils) groups, n=207 pupils) 


Withdrew (n=1 year group, n=30 pupils) Withdrew (n=1 year group, n=30 pupils) 


Analysed (n=5 year groups, n=205 Analysed (n=4 year groups, n=167 pupils) 
pupils) 


Excluded from analysis (n=2 year groups, 
Excluded from analysis (n=1 year group, n=70 pupils) 


n=30 pupils) 
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Outcomes and analysis 
Main analysis 


The aim of this element of the evaluation was to provide initial quantitative outcome data on the 
efficacy of the programme through the analysis of the quantitative data collected on attainment and 
affective outcomes at the end of the programme. 


Ordinarily in a matched pairs design, data from compromised pairs is removed from the analysis. In 
this study two of the three pairs were compromised due to one school withdrawing at the start of the 
study and another school withdrawing before the control data was collected. Conducting the analysis 
with only the remaining intact pair would make it practically impossible to draw any reliable, unbiased 
conclusions regarding the potential impact of the programme. The research team decided that on this 
occasion ‘breaking’ the matching was justified because, whilst attrition introduces a high risk of bias 
this is somewhat attenuated (but certainly not eliminated) as: 


1. the intraclass correlations were very low (0.01—0.04) indicating that the year groups and 
schools were reasonably similar to each other; 

2. the reasons for withdrawal appeared to be unrelated to the outcomes; and 

3. each school contained within it both intervention and control groups. 


Given this, it was considered meaningful to include all the data in the analysis whilst acknowledging a 
considerable caveat—the evident threats to the internal and external validity of the study and the bias 
this would introduce. 


The tables below present the post-test differences between the intervention and control groups on the 
outcome measures adjusted for the effects of gender and FSM eligibility, as well as taking into 
account any clustering at the level of the year group. As different measures were used across 
outcomes, the scores for each measure were standardised prior to combining for the purposes of 
analysis. The raw post-test scores for each measure (broken down by intervention and control group) 
can be found in Table 7 in Appendix 4. 


The dependent variable and covariates (gender and FSM) were standardised prior to analysis with a 
mean of zero and a standard deviation of one. The analysis was based on a complete case analysis 
and was performed using a clustered sandwich estimator. Effect sizes were calculated using Hedges’ 
g. The full models containing all the parameter estimates for each model are reported in Appendix 4. 
Missing data from the 40 pupils in the control condition of the school that withdrew mid way through 
data collection were not imputed. Unfortunately there is no data available with which to compare the 
characteristics of these children to those for whom post-test attainment data was collected. 
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Table 6: Summary of main effects at post-test” 
Adjusted post test mean Intraclass Effect size 
Outcome Intervention Control correlation 95% Cl 
Veta €3)) WV Eeeta (1 )) coefficient 
Numeracy -0.02 (1.01) 0.02 (0.98) 0.05 -0.04 p=0.81 
[-0.37, 0.28] 


Literacy -0.01 (1.04) 0.01 (0.95) 0.05 -0.01 p=0.93 
[-0.32, 0.30] 

Attitude towards -0.05 (1.00) 0.11 (0.98) 0.00 -0.16 p=0.22 

school [-0.37, 0.06] 


The results indicate that in this sample of pupils there were no differences between the intervention 
and control groups in relation to the primary outcomes (attainment in literacy or numeracy) with effect 
size estimates close to zero in both cases. With respect to the secondary outcome ‘attitude to school’, 
the effect size estimate is larger (ES = -0.16) indicating that the intervention group are reporting that 
they enjoy school less than the control group. This result is not statistically significant. 


Exploratory analysis 


Due to the small numbers of FSM pupils it was not possible to conduct the planned sub-group 
analysis exploring the impact of the intervention on this subset of pupils. 


An exploratory sub-group analysis was conducted, however, to explore whether the intervention 
worked differently for boys or girls. This was done by creating an interaction term between the ‘Group’ 
and ‘Gender’ dummy variables (where boy = ‘1’, girl = ‘0’; intervention = ‘1’ and control = ‘0’) and 
incorporating this term as an additional covariate in each of the three regression models for literacy, 
numeracy and attitude. 


There appeared to be a small differential impact of the programme suggesting that the programme 
worked better for girls with respect to both literacy and numeracy. In relation to literacy, girls in the 
intervention group performed the best overall compared to girls in the control group and compared to 
boys overall (Beta (interaction term) = -0.18, p = 0.02) (Figure 2). 


Figure 2: Adjusted post-test scores for literacy, by gender* 


0.30 


Intervention 
™ Control 


Post-test literacy scores 


Gender 


*Scores standardised so that the sample as a whole has a mean of zero and standard deviation of one. 


” See Tables 8, 9 and 10 in Appendix 4 for details of the fitted models. 
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With respect to numeracy, girls in the control group performed worst compared to girls in the 
intervention group and boys overall (Figure 3) (Beta (interaction term) = -0.13, p=0.06). These effects 
were small however and there was no evidence of an interaction effect with respect to the secondary 
outcome, attitude to school. 


Figure 3: Adjusted post-test scores for numeracy, by gender* 
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Gender 


*Scores standardised so that the sample as a whole has a mean of zero and standard deviation of one. 


Cost 


The cost of the programme as delivered in the one-year pilot study was estimated at £9.08 per pupil. 
This estimate is based on 237 pupils receiving the programme, this being the first year of 
implementation, and includes two initial half-day training sessions. These sessions were provided by 
an external trainer, costing £2,150 in total. This estimate does not include direct salary costs, supply 
cover for training, or the out of school time required for teachers to develop the necessary resources. 
Once teachers have been trained to deliver the programme there are no on-going costs to the school. 
On this basis, the estimated cost of implementing the programme over three years works out at £3.02 
per pupil per year. 
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Process evaluation 


This section presents the findings from the qualitative process evaluation, which involved interviews 
with teachers and headteachers, focus groups with pupils, and observations of lessons in each of the 
five participating schools. 


Training 


Overall, teachers were positive about the training and guidance they received on implementing PAL. 
All intervention teachers were enthusiastic and optimistic about implementing the programme 
following the initial training event in January 2014. By the second training event in March 2014, only a 
few teachers remained enthusiastic with over half the teachers finding PAL in the real-life setting of 
the classroom a struggle. . 


On the whole and at the outset, teachers were very positive about PAL, although they did report 
having a number of apprehensions prior to the training. There were two key issues raised by 
teachers: (1) how the programme would be fitted into the timetable, and (2) creating activities. The 
majority of teachers reported that the first training session was ‘good’ and not only helped to alleviate 
any initial concerns, but also provided ‘a few great ideas’. One teacher did feel that they could have 
been provided with more ‘ideas’ during the training but acknowledged that being required to come up 
with their own activities and lesson plans allowed them to ‘own’ the programme. Below is a selection 
of comments that are broadly representative of the teachers’ views after the initial training: 


‘l was just a bit confused and very apprehensive before the training. | wasn’t sure whether it 
was part of my hour literacy lesson or whether | was doing it in addition. That was what my 
biggest fear was—when | was going to fit it in. So just having the training and seeing how 
quick and how relevant | can make it to every lesson—that completely put my mind at rest.’ 


Most teachers suggested the second training event was an opportunity to reflect, discuss and share 
the challenges of implementing PAL in its first month. Teachers were relieved that they were mostly 
encountering the same difficulties. Overall, regardless of the quality or inconsistency of PAL 
implementation across the schools, the majority of the teachers did not relate this to the training they 
received but to reasons that will be discussed in the ‘barriers to delivery’ section below. 


‘It was nice to be able to hear what other teachers were worried about because | was 
panicking that | wasn’t doing it for long enough, or at the right time, or that | wasn’t doing it 
properly, so it was nice to get together and share, and obviously having them telling us that, 
yeah, what we were doing seems fine.’ 


Developing the physical activities 


For this intervention, classroom teachers were required to integrate physical activity with existing 
lessons linked to specific academic learning objectives for numeracy and literacy. The physical 
activities were to be delivered intermittently throughout the school day (preferably one 15-minute 
session in the morning and [one in] the afternoon) to promote 90 minutes per week of moderate to 
vigorous physical activity. This was not a manualised programme and so, in addition to the suggested 
physical activities from the training event, there was a bottom-up approach to programme 
development whereby teachers were encouraged to develop their own activities. With this in mind, the 
programme developers and the TDT intended to facilitate a process of teacher-led collaborative 
enquiry through face-to-face meetings as well as through an online portal where activities and 
resources could be shared. The majority of teachers used the suggested activities from the training 
and adapted these to suit their own lessons as the school year progressed. 
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‘1am doing both, sort of... | am using some of the activities that were spoken about at the 
training and then | have also changed some of these to suit my lessons or the children or 
even sometimes just the classroom. If it’s a good day | might take them outside and make the 
exercise a little more vigorous.’ 


One teacher reported that she did make use of other teachers’ ideas, and described this as ‘naughty’ 
when in fact it was the desired intention of the programme developers that activities and resources 
should be developed and shared. 


‘| have been using either activities that were suggested at the training or I’ve been a bit 
naughty and I’ve been checking other people’s and borrowing their ideas [laughs].’ 


Integrating PAL into curriculum objectives 


Two main reasons teachers struggled with developing their own activities were to do with finding it 
‘hard to be creative’ and, relatedly, linking the activities to curriculum objectives, especially with 
respect to literacy. Some teachers felt that they were ‘regurgitating’ the same activities in class: ‘they 
do a lot of star jumping to times tables’. One teacher came up with the idea of asking children to 
‘come up with some ideas, because they quite like it’. 


While teachers expressed a clear desire to implement the programme, they needed it to be ‘relevant 
to what [the class was] learning’. Developing activities and integrating PAL into literacy lessons 
appeared to be particularly ‘difficult’ and time consuming. Furthermore, it seemed to interfere with 
some teachers’ continual assessment of their pupils’ progress such that they were unable to clearly 
see ‘who is really learning and who is not’: 


‘If | say, ‘right, okay we’re going to stretch up when you hear an adverb,’ and | can see some 
of them looking round, | don’t know whether that’s because they’re thinking slower, they're 
copying or whether they don’t know. | have no idea, it’s too difficult to judge and measure. ’ 


Barriers to delivery 


Teachers reported a number of barriers to delivering the PAL programme and these fell broadly into 
three categories: (1) time constraints related to preparation and delivery, (2) the suitability of the 
programme for all children in the class, and (3) space constraints within the classroom setting. 


Time constraints 


All teachers interviewed referred to time constraints being a major challenge to successful 
implementation. Three particular areas were identified where time pressure relating to programme 
implementation was an issue: 


1. creating and developing new activities and lessons; 
2. implementing the programme in the classroom; and 
3. sharing and uploading resources to the online portal. 


It was acknowledged that because the programme was in the development phase, more time was 
also needed to be creative and that possibly after first year of implementation teachers would become 
more comfortable with activities therefore requiring less time. The importance of this is reflected in the 
comments below: 


‘This is new to all of us, not just the children. It takes time to create the lessons as well as 
carrying them out. | know this would be so much quicker next year because we would be 
more familiar and we might even have a manual.’ 
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‘Well, lots of people were saying, including myself, “it’s time to put it on the portal”. This week, 
| haven't added anything, or last week. | have been doing it, mostly, but | just haven’t had time 
to put it on the portal.’ 


As noted, having implementation teachers share their [ideas and] activities on the online portal was 
fundamental to the development of the programme, particularly by adding newly developed activities 
to create a rich and valuable resource that all teachers could access and use. As detailed in the 
fidelity section below, there was variation across the participating schools in their adherence to this 
aspect of implementation. The majority of teachers did not use the website to access ideas from other 
teachers and cited time constraints as the reason. Only one out of seven teachers uploaded the 
activities as recommended (a full and comprehensive list of adapted and developed activities 
uploaded at the end of every week) with a few teachers sporadically uploading what they had 
implemented (activities uploaded at the end of the month or at the end of the school year, and mainly 
just the numeracy activities). The majority of teachers, however, abandoned this activity after the first 
month of implementing the programme. All teachers considered lack of time to be a real challenge in 
the successful completion of this component of the programme. 


On the second point relating to time, some teachers worried that 90 minutes per week was a ‘large 
chunk of their learning time’, so knowing that the intervention had robust outcomes—either health or 
academic—was important: 


‘Sometimes what you need to do with them is a sitting activity and showing the whiteboards, 
that sort of thing. And what | don’t want to do is get them jumping up and down doing 
something if that’s not the best way for them to learn’ 


Similarly, another teacher stated: 


‘1am always under pressure to increase the attainment for all the children in my class and it’s 
conflicting with that because it’s slowing things down.’ 


Programme applicability 


Teachers identified a second barrier that pointed to the programme not suiting all pupils. A few 
teachers suggested that the pupils attending their schools were mainly ‘high achievers’ and had 
possibly less to gain and more to lose by taking part in this kind of programme where the benefits 
were not yet clear. Concerns were raised that the programme might be a ‘distraction’ for some 
children who need a ‘more structured environment’ in which to learn. As one teacher commented: 


‘| think it doesn’t suit all children in the class, particularly the maths section... there’s that little 
bit more freedom, and they just push it a little bit too far sometimes. ...be silly, rather than... 
It’s quite difficult. | think that’s probably one of those things that we'd have to think about 
moving into next term, when we’re doing it full-time, is how we’re going to manage children 
that it isn’t helping.’ 


Some teachers ‘struggled’ to see the benefits for their pupils because they were affluent, already very 
physically active, and often took part in after school sports programmes: 


‘| think if you could take this into an inner city school that has a small playground, | think the 
benefit would be a lot more obvious and beneficial that it is to our children. Because basically, 
we find they come in from break or play, because normally, we do it straight at the start of the 
lesson. They’ve been running round. They are hot. They are sweaty. They all moan, and then 
you Say, “Right, on your feet,” and they go, “Oh no”.’ 
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Space constraints 


Finally, the majority of the teachers were concerned about space constraints in the classroom and the 
need for more lessons examples that could be taught within a small classroom, to avoid repetition and 
restriction of activities. This is illustrated in the following comments: 


‘Well, limitations, within the classroom space, we really are doing things like jumping on the 
spot, running up and down, a few stretches, simple moves, like we do some karate stuff.’ 


‘The size of the classroom limits. Yes, | think that is a thing that limits us in terms of often 
having very same... the same activities because we are restricted. ' 


Acceptance of the programme 


The programme was well received by all five schools taking part in the pilot study and, importantly, 
had the support of the school headteacher at the outset to implement PAL. However, evidence from 
the final interviews with the intervention teachers suggests this enthusiasm diminished in most 
schools after a few months of implementation for reasons mentioned above (see ‘Barriers to 
implementation’). A common reflection among teachers was that the development of PAL became 
one dimensional—using existing material and not having time to action other aspects of the 
programme, for example, developing new materials and sharing on the online portal: 


‘1am managing to carry out the physical activities mostly with maths lessons, seldom with 
literacy work now and | really just don't have time to think up ways to make it work with 
literacy. | keeping meaning to look on the portal for ideas, but again time is always against 
me.’ 


Pupils were also excited and enthusiastic about the programme at the outset of implementation, as 
reported by their teachers. Again, most teachers suggested that pupils became bored with the 
programme after a few months and demonstrated this by their lack of keenness at the start of the 
lessons, as reflected in this comment: 


‘The children started off really positively. I'd say that’s slipped in the last couple of... maybe 
the last one or two months—they’ve not been quite so keen. But | couldn’t quite judge 
whether it’s just because it’s the end of Year 5, or because they’re bored of it, or... | can’t 
quite judge without talking to them specifically. Sometimes there would be a bit of a groan, 
which I was quite surprised with, because, to begin with, they were really excited about it all.’ 


In contrast, the overwhelming majority of the children from all of the focus groups were very positive 
about PAL. The children referred to the programme as ‘much more fun’ and enjoyable than other 
types of lessons. They thought it was ‘very clever keeping fit and active during your lessons’ and 
helped them ‘to remember things better’. As illustrated below by a representative sample of 
comments taken from across the focus groups, the pupils were keen to convey their enthusiasm for 
the programme: 


‘Yeah, it’s a better way to do it, and it’s a more fun way to do it. So sitting there isn't that fun, 
but like, you want to have fun in a lesson, and that's the way to do it.’ 


‘Because it’s the way to get kids more fit, instead of just doing normal lessons and they can 
learn too.’ 


‘It makes the classroom more happy.’ 
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A few children commented that they didn’t enjoy taking part in PAL. These children pointed to PAL 
being ‘boring’ with too much repetition of certain activities such as star jumps. A few other children 
reported that they found the lessons very tiring. Comments included: 


‘A bit boring, because you're doing the same thing all the time.’ 


‘Well sometimes when it’s a really, really hot day like today you just get a bit worn out and 
you're tired and you don’t really pay attention. ’ 


Unintended consequences of programme 


Teachers reported that the PAL could have implications for some pupils in terms of behaviour. 
Several teachers discussed how classroom behaviour would deteriorate for some pupils as they used 
it as an excuse to ‘act silly’ and this reflected in these participating pupils being unable to concentrate 
and focus on the lessons. 


‘Because although we go straight out onto the field or playground, of course, as soon as you 
start taking children out, they just go a bit mental, and you want them calm to teach.’ 


‘Some of the children get absolutely hyper. You don’t really want that at the start of a maths 
lesson. You want them focused, so...’ 


Fidelity 


PAL was not a manualised programme and therefore accurate fidelity is difficult to gauge. All schools 
participated as agreed with all teachers attending initial training session. No teacher was able to 
incorporate 90 minutes per week of PAL within existing literacy and numeracy lessons. Four out of 
seven teachers reported that they were able to incorporate 15 minutes of PAL into their daily 
numeracy lesson. It seemed more challenging to incorporate PAL into literacy lessons with teachers 
reporting that they managed to do so occasionally—up to three days per week. All seven teachers 
managed to integrated the PAL with mathematics most days, with some teachers having more 
difficulty than others: 


‘Every day, unless we’re doing something in particular which means we have to have the 
whole lesson on something else. So, for example, we had assessments recently so | had to 
cancel. It’s always maths. I’m trying to do a few literacy as well, if | can.’ 


The most that any teacher was able incorporate PAL with the literacy lessons was three times a 
week. 


‘We are attempting, and I'll say attempting... to plan for literacy and maths every day. I’d say 
that we definitely get the maths one done every day. And the literacy one, I’d say three out of 
five, probably, just because it involves so much more thinking and we are so under pressure 
for the children to reach certain levels we have to make sure that what they are learning is 
has an outcome for them.’ 


The teachers who managed to integrate the physical activity with literacy indicated that it took a 
considerable effort, and although they did incorporate it, it was not to the level that was recommended 
and was very much on an ad hoc basis: 


‘Yeah, we fitted it in more with mental maths type activities, whereas we're finding it difficult to 
be able to find something relevant to do with it for English or for the foundation subjects.’ 
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Perceived outcomes of the intervention 


When asked about the perceived outcomes from PAL for the pupils involved, the overwhelming 
majority of teachers made reference to increased physical and health benefits, stating that it ‘was 
good for their fitness levels’ and that this would improve ‘over time’. 


Some of the teachers related the gains/outcomes of the programme to greater enjoyment of lessons 
for the children, particularly at the beginning of the programme when it was new for pupils to learn 
their lessons in this way. 


In relation to the possible academic benefits of the programme, however, most teachers were 
sceptical and ‘struggled to find benefits’. Instead, some suggested that it was a potential distraction 
and one teacher was ‘unsure’ what they should even be looking out for: 


‘| think this programme could be a distraction to the children learning their maths or English, 
instead of being a benefit.’ 


A few teachers suggested that they would find it difficult to see outcomes for the children for reasons 
suggested in the previous section—mainly that the programme was just not really suitable for some 
children. 


A key observation expressed by the majority of teachers was that it was hard to measure the learning 
output from the programme because of the difficulties of linking activities to the expected learning 
outcomes of the curriculum. The importance of this is reflected in the comment below: 


‘In a normal teaching situation, we know exactly what to expect from all our children. But it’s 
hard to measure whether, from our point of view, because you've got to keep an eye on all 
the children, to make sure they're being sensible, they're being safe.’ 


Interestingly, the majority of children across the focus groups perceived the outcomes from the PAL 
as helping to prepare their brain for learning, making it ‘much easier to learn’. Children related better 
blood flow to the brain and being physically fit as complementary to learning. As illustrated below by a 
representative sample of comments taken across the focus groups, the children were keen to convey 
how this was the case: 


‘| think it’s also your heart beats harder and you get more blood into your brain so you can 
think better.’ 


‘Yeah because like your brain’s working more and it absorbs because if you’re tired your 
brain’s just thinking about, “I’m tired”, and when you're doing it you get more active and then 
your brain starts remembering what you're doing in the lesson.’ 


A few children reported that PAL helped them to feel ‘more involved’ in their lessons than before, and 
this had positive benefits for them in terms of, for example: better recall, improved concentration, and 
aptitude to work harder. This is reflected in comments such as: 


‘| think it helps me with my learning because you just get more involved with the lesson and it 
makes you work a bit harder.’ 


‘| think | concentrate more.’ 


‘Yeah, it helps you a bit to remember stuff.’ 
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Formative findings 


Overall, in this first year of implementation, headteachers, teachers and pupils enthusiastically 
embraced the PAL programme. Implementation was not without its challenges however and there 
was considerable variation in programme delivery across the five schools. The process evaluation 
has highlighted inherent issues in adopting a programme during its developmental phase. 


The initial ‘bottom up’ approach of supporting teachers to build on the training they had received, and 
to develop their own activities and resources, encouraged flexibility, experiment and local ownership. 
The original intention of the programme developers was that each teacher would share their newly 
developed resources with others (through the online portal) to create a rich and valuable resource that 
all teachers could access and use. The downside of this approach, however, was that teachers found 
this challenging, reporting that they did not have enough time—or felt they lacked the necessary 
creativity—meaning that the online resource did not grow or develop, as was the original intention. 
Similarly, the integration of physical activities into lesson plans wasn’t always straightforward or easy, 
and this was especially the case with respect to literacy lessons. A consequence of all this for some 
teachers was that the programme became a burden to implement and as a result enthusiasm waned. 


Aligned to this bottom-up approach to programme development, an additional intention of the 
programme developers and the TDT was to facilitate a process of teacher-led collaborative enquiry 
through face-to-face meetings as well as through the online portal. This support mechanism is 
important in order to find the right balance between fidelity and local adaptation. However, this 
endeavour did not take place in the end, and while teachers may have benefitted from additional 
support throughout the delivery phase, it may well have been time-intensive and thus problematic 
given the existing demand on time that teachers reported. In supporting teacher collaborative inquiry it 
is recognised that teachers have the ability to drive and initiate change, but it is often the case that 
complex layers of support are required to achieve this goal (Nelson and Slavit, 2008) including 
fostering school cultures, structures and processes that are conducive to supporting teacher 
collaborative enquiry. Although training needs were met and teachers did not complain about poor 
support from either the teacher-led collaborative enquiry or colleagues through the online portal, from 
an evaluator’s observational perspective it seems likely that this was one of the factors that caused 
the intervention to be poorly executed resulting in levels of fidelity deteriorating over time in some 
schools. 


Finally, teachers expressed the view that they would like to be reassured that the programme 
activities and outcomes are consistent with the curriculum aims—that the programme is a benefit 
rather than a distraction. The existing pressure on teachers to raise achievement, combined with the 
self-directed nature of implementing and developing a programme they perhaps didn’t quite believe 
in, may have been sufficient to tip teachers into apathy after an enthusiastic start. The implications of 
these findings for developing and improving the programme are discussed further in the following 
section. 


Control group activity 


The process evaluation did not indicate any spill-over effect to control group pupils or any 
resentfulness or rivalry between the intervention and control groups as evidenced by both the 
intervention and control teachers. However, the small scale of the process evaluation did not allow for 
data gathering from control group pupils so we are unable to make claims about their perceptions of 
the trial; interviews with the control teachers indicated that their pupils did not adopt or know very 
much about the intervention and therefore it is unlikely that they adopted any components of it. 
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Conclusion 


| Key Conclusions 


1. Evidence for the feasibility of implementing Physically Active Lessons in the classroom is mixed; 
both teachers and children reported enjoying the programme, however, the programme created a lot 
of additional work for teachers and thus, enthusiasm tended to fade towards the end of the study. 


2. Teachers found the training enjoyable and helpful, however, they reported that it was sometimes 
difficult and often time consuming to develop lesson plans and think of ways to integrate physical 
activity into lessons, especially literacy lessons. 

3. More work is needed to develop the programme before it is ready to be tested using a full trial. 
Additional work is required to develop a programme manual and integrate additional support 
mechanisms for teachers throughout the delivery period of the intervention. 

4. If the programme is taken to a full effectiveness trial it is recommended that paper-based tests are 
used, additional explanatory variables are measured, and randomisation can successfully occur at the 
year-group level. 


Limitations 


This was a small pilot study and was not designed as a fully powered efficacy or effectiveness trial. 
Despite this we have attempted to provide some indication of effect by analysing the outcome data 
that was collected. However, without a measure of pre-test performance included in the model, the 
proportion of variation in the dependent variable that the regression models explain is very small (less 
than 3%). 


There were several threats to the internal validity of the trial which means that the findings are at a 
high risk of bias and certainly imprecise (indicated by the wide confidence intervals). The sample size 
was small, different tests were used to measure the outcomes, the data collection was staggered and 
we cannot be sure that differential attrition and missing data has not further biased the results. There 
were also threats to the external validity of the study and so it is not possible to generalize the results 
to the wider population: the sample was non-random, drawn from one local authority area, and 
participating schools were not representative of all primary schools, for example, having lower 
proportions of pupils with special educational needs, with English as an additional language, and free 
school meal eligibility compared to other state primary schools in England. 


Interpretation 


The results relating to the programme’s impact on literacy and numeracy outcomes should be 
interpreted with a good deal of caution: the conclusions we can draw are limited. Whilst this study has 
not been able to detect programme effects this is not in any way indicative of the true potential effect 
of PAL and it needs to be tested rigorously prior to drawing any conclusions with respect to this. 


The evaluation consisted of two components related to (1) the feasibility of implementing the 
programme, and (2) the feasibility of designing and conducting a large scale RCT in the future. The 
evidence for whether the approach is feasible or not is mixed; both teachers and children embraced 
the programme enthusiastically, however the programme required a lot of additional work for teachers 
and consequently teacher enthusiasm tended to fade. With respect to the feasibility of a large-scale 
RCT, more work is needed to develop the programme before it is ready to be tested using a full trial. 
The current study has generated important insights into some of the issues associated with the 
development and first-year implementation of the Physically Active Lessons programme, and these 
are summarized in more detail below. 
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Learning from the process evaluation 


The aim of the process evaluation was to inform and support the development and adaptation of the 
intervention for an English context through independent observations of programme development, 
teacher training, and programme implementation. All teachers and pupils embraced the programme 
with enthusiasm at the beginning, however, as the school year progressed enthusiasm started to 
dissipate. The difficulties that arose can be condensed into two main themes: 


1. There was no clear set of instructions or support in relation to how to develop and implement 
the programme. This resulted in inconsistent programme delivery across all five schools and 
meant that there was no clear benchmark against which to measure fidelity. It also meant that 
at times teachers were: 

¢ unsure whether they were delivering the programme correctly; 

e — finding it challenging to integrate activities into lesson plans (especially in the case of 
literacy); and 

¢* unclear as to whether the programme was achieving the intended outcomes, or 
whether it was successfully contributing to curriculum learning objectives. 


2. There was limited available teacher time to: 
* create and develop new, appropriate activities and lessons; 
¢ implement the programme in the classroom; or 
e share and upload resources to the online portal. 


How the programme could be improved 


Many of the issues summarised above can be addressed through the development of a programme 
manual alongside additional mentoring and support. Indeed, this step is extremely important if the 
programme is to be taken to scale. 


Such a manual would provide potential users with a clear understanding of the theoretical 
underpinning of the programme, it’s aims and outcomes, as well as a comprehensive guide on how to 
implement it. A manual would help to streamline the programme to ensure that it is both time efficient 
for teachers and has the maximum potential of achieving its outcomes. Specifically, the development 
of such a manual should take the issues raised above into careful consideration and include: 


¢ A theoretical framework that underpins the programme and supports its intended outcomes. 
The relationship between the programme's activities, outcomes, and the curriculum for both 
literacy and numeracy should be clearly described and evidence informed. 

¢ Arrange of physical activities that are appropriate, well-specified, varied, and fully developed. 
Activities should be linked to the underlying mechanisms of change as well as the programme 
outcomes. These should include activities that are appropriate for indoor and/or outdoor use 
as well as activities that can be used in small classrooms with limited space. The existing 
activities developed by the teachers in this study would be a good starting point in this regard. 

¢ Guidelines on how various activities can be better integrated into numeracy and especially 
literacy lessons. 

¢« A transparent benchmark that describes what full compliance/implementation should look like. 
This will also make fidelity and quality of implementation easier to measure during evaluation. 

* Supplementary guidance and advice regarding strategies teachers can use to avoid 
disruption or unsettling students immediately after physical activities. 

¢ A stronger mentoring and support structure throughout the delivery phase to ensure fidelity 
and quality of implementation. 
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Learning from the pilot trial 


The aim of the pilot trial was to: inform the selection of outcomes and measures; determine the 
feasibility of the randomisation process; provide initial quantitative outcome data on the efficacy of the 
programme; and provide advice on the requirements and process for undertaking a full scale RCT. 


Selection of outcomes and measures 


It emerged that, compared to the digital tests, the paper and pencil tests were much easier to 
administer in this sample of primary schools. Teachers expressed a clear preference for the paper- 
based tests: schools did not always have adequate equipment for digital testing—or access to the 
necessary technical expertise—which exacerbated the complexity of administration. It resulted in 
delays in data collection and contributed to the disenfranchisement of teachers and headteachers. 
Schools did not raise any issues with respect to the content of the tests, and both were deemed 
suitable. The scoring of the digital tests was much more straightforward compared to the paper tests 
that were hand scored by the research team. This would be a time-consuming process in a larger trial 
and the related costs of test scoring (a service that the publisher can provide) should be considered 
for any future trial. 


The remit of the evaluation team was to collect outcome data relating to attainment and enjoyment of 
school, however the mechanism by which physical activity is related to cognitive gains is not well 
understood. For this reason, future trials should seek to identify and measure plausible explanatory 
variables which are suggested by the literature and support the programme’s underpinning theory of 
change. Such variables might include those related to health, well-being, and self esteem as well as 
to cognitive skills such as memory, concentration, and self-regulation. 


Feasibility of the randomisation process 


The unit of randomisation was the year group. This appeared to work well, and there was no evidence 
of any spill-over effects into the control group. There were no other issues related to the 
randomisation process, and allocating year groups rather than schools to the intervention meant that 
all schools had the opportunity to deliver the intervention to one cohort of their pupils. Even though 
one school dropped out, this genuinely seemed to be unrelated to the intervention or its outcomes. 
Attrition is also an issue for studies of this nature and whilst it can rarely be completely avoided steps 
can be taken to prevent it occurring, including building good relationships with schools, ensuring 
headteachers and teachers are aware of what involvement will mean for their school, and by asking 
schools to sign a memorandum of understanding which makes clear the expectations on both sides. 


Initial quantitative outcome data 


The results from this pilot trial should be treated very cautiously for the reasons outlined above. There 
were no differences between the intervention and control groups regarding either literacy or 
numeracy. The control group reported greater enjoyment of school than the intervention group 
however this difference was not statistically significant. There was a suggestion that the programme 
might work better for girls rather than boys. However, to be considered practically meaningful, all of 
these findings would need to be properly tested via a large scale RCT. Due to the low proportion of 
disadvantaged children in the sample it was not possible to explore whether the programme might 
work better for children from deprived backgrounds. Given the EEF’s focus on improving outcomes for 
disadvantaged children and young people, it is important that future trials are designed to test this. 


Education Endowment Foundation 30 


Physically Active Lessons 


Future research and publications 


A sufficiently powered efficacy trial is necessary to provide a more equitable test of the likely impact of 
the programme allowing more robust conclusions to be drawn regarding its effectiveness in improving 
academic attainment. Additional questions should include: 


e What is the pathway through which the programme exerts its influence (if any) on academic 
outcomes? 


e Does the programme suit certain groups of children better than others, for example, boys or 
girls, low achieving children or children from deprived backgrounds? 


e Are some school contexts better suited than others to implementing the programme, for 
example schools with more or less physical classroom or outdoor space? 
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Appendix 1: Memorandum of understanding 


Physically Active Lessons Evaluation 
Memorandum of Understanding 


This memorandum of understanding is between the school taking part in the research (named below) 
and the research teams located in University of Bristol and the Centre for Effective Education in 
Queen’s University Belfast. 


Name of School: 


Its purpose is to clarify the roles and responsibilities of each organisation to facilitate the smooth 
running of the research project. 


To receive the Physically Active Lessons programme and take part in this study please read and sign 
this Memorandum of Understanding. 


Background 


The Centre for Effective Education at Queen’s University Belfast have been asked to carry out an 
independent evaluation of the implementation of the Physically Active Lessons programme in six 
schools. This research aims to find out what headteachers, teachers and pupils think of the 
programme, what were some of challenges and successes that were experienced and whether taking 
part in the programme has helped to improve children’s literacy and numeracy. 


The research team will ensure that parents are informed of the research and that opt-out consent is 
obtained for their child’s participation. 


Agreeing to take part in the research will mean the following: 
1. Random allocation of Year Group to get the programme 


Each participating school will put forward both their Year 4 and Year 5 year groups to be included in 
the research. The CEE research team will randomly allocate one of the two year groups to get the 
programme. The teachers of the selected year group (also called the intervention group) will receive 
the training in January 2014 and deliver the programme between February and July 2014. 


The year group that is not selected to get the programme will act as a comparison group (also called 
the control group). Teachers of this year group should only deliver the regular curriculum i.e. 
‘business as usual’. 


2. Teacher training 


Teachers of the classes in the intervention group will attend one day’s training in January 2014, 
delivered by the Teacher Development Trust. The research team will attend and observe the teacher 
training and will provide teachers with a more in depth description of the study. 


3. On-going teacher support 


The Teacher Development Trust will provide on-going support for teachers throughout the duration of 
the programme. This will be in the form of teacher led collaborative enquiry and the research team 
will attend some of these meetings/sessions in order to better understand this element of 
implementation. 


Education Endowment Foundation 34 


Physically Active Lessons 


4. Teacher telephone interviews 


The research team would like to interview teachers in the intervention group by telephone when they 
have completed the training in order to explore their initial perceptions of the programme (January 
2014) and again one month into implementation (April 2014). These telephone interviews will take 
between ten and fifteen minutes. 


5. School visits 


The CEE research team will visit each participating school for a day in May 2014 and again for a day 
in July 2014. They will liaise directly with each school about the timing and nature of these visits so 
that they occur at a time most convenient to the school and cause as little disruption as possible. 


During the visit in May the research team will: 


¢ Interview teachers who have taken part in the programme as well as those who have not (30-40 
minutes) 


¢ Interview headteachers (30-40 minutes) 
¢ Undertake a lesson observation in which the programme is being implemented 
¢ Conduct a focus group with 6 to 8 children who have taken part in the programme 
During the visit in July: 
¢ The research team will administer literacy and numeracy tests to all Year 4 and Year 5 children 


¢ In order to determine which mode of testing is the most practical in the future, half of the six 
schools will be tested using paper tests and the other half will be tested using digital tests 


¢ This testing will be organised and administered by the research team who will liaise directly with 
each school and will personally visit each school to undertake the testing on a date that best 
suits the school 


¢ Schools will be provided with the results of the tests for their own use 


A researcher from the University of Bristol will also visit the school once every half-term. Again, the 
timing will be agreed with the school and teacher to ensure disruption is minimized. 


During these visits, the University of Bristol researcher will: 
¢ Observe one Physically Active Lesson and one regular lesson 


¢ Collect data on children’s ‘Time on Task’. This simply involves the researcher noting how long 
each child is concentrating for 


e« The results will be shared with schools for their own use 
6. Addition pupil information 


The school will provide the CEE research team with the UPN for each child, name, date of birth and 
postcode. This information will enable the research team to link to the National Pupil Database 
records. 
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Will participation in the study be kept confidential? 


Pupils’ test responses and any other pupil data will be treated with the strictest confidence. The 
responses will be collected by the research team. Named data will be matched with the National Pupil 
Database and shared with University of Bristol, the Centre for Effective Education (the research 
team), the Education Endowment Foundation and the UK Data Archive for research purposes. No 
individual school or pupil will be identified in any report arising from the research. 


| understand that participation in this programme and evaluation is entirely voluntary and | am clear 
about my school’s role in this research study. | have been given the opportunity to ask questions and 
any concerns that | may have raised have been addressed. 


Signed Date 


Position 
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Appendix 2: Parent consent letter 


Physically Active Lessons Evaluation 
Parent Information Sheet and Consent Form 


Hello, 


We are a research team from the Centre for Effective Education at Queen’s University Belfast. We 
and the University of Bristol are carrying out a study in your child’s school and would like your child to 
take part. Before you make a decision it is important for you to know why the research is being done 
and what it will involve. If you do not want your child to take part, we will ask you to sign the attached 
consent form and return it to your child’s teacher. 


Please take your time to read the following information carefully and discuss it with others if you wish. 


Why are we doing this study? 

This year your child’s school is rolling out a new programme called Physically Active Lessons. The 
programme involves using short bursts of physical activity (15 minutes long) in the classroom, such as 
hops and starjumps. It is thought that doing this can help children to learn better as well as improve 
their phsycial fitness. The reason we are doing this study is to find out whether Physically Active 
Lessons really does help children to learn and whether children enjoy learning in this way. The study 
is being funded by the Education Endowment Foundation. 


What will happen if my child takes part? 

If your child takes part, their Year Group will be randomly chosen to either get the programme 
(Physically Active Lessons) or to continue with the normal delivery of lessons. If your child’s year 
group is selected to get the programme this will involve the class teacher using short bursts of 
physical activities which last 15 minutes, twice a day for the remainder of the school year. The 
activities can be done in the classroom and children can do them in their school uniform. 


During the spring and summer terms the research team will monitor children’s levels of concentration 
and at the end of the summer term, in July 2014, children (both those who received the programme 
and those who did not) will be asked to take a literacy and numeracy test. The tests will be delivered 
by the research team and pupils’ test responses and any other pupil data will be treated with the 
strictest confidence. 


We may also ask your child to take part in a short focus group with some of their classmates to ask 
them what they thought of the programme and whether they enjoyed doing physical activities as part 
of their lessons. 


Named data will be matched with the National Pupil Database and shared with University of Bristol, 
the Centre for Effecive Education, the Education Endowment Foundation and the UK Data Archive for 
research purposes. We will not use your child’s name or the name of the school in any report arising 
from the research. 


Does my child have to take part? 

You are free to agree for your child to take part in one, none or all parts of the study. If you do NOT 
want your child to take part in one or any part of the study, please inform their teacher using the form 
below. We expect that your child will enjoy doing the tests and being part of the programme. Your 
child may withdraw or decide not to take part at any time and we will explain this to your child. 


How long will the study last? 
This study will begin in January 2014 and will finish in July 2014. 


What if | change my mind? 

You can change your mind at any time during the study and decide that you do not want your child to 
take part anymore in either the focus group or the tests. You do not even have to give a reason why. 
Your decision to take part will in no way impact on your relationship with any of the organising bodies 
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(Queen’s University Belfast, University of Bristol and the Teacher Development Trust) now or in the 
future. 


Will our participation in the study be kept confidential? 


All information that is collected about you and your child during the course of the research will be kept 
strictly confidential. Research folders will be kept in a locked office at all times. Access to these 
folders will be restricted to study investigators and statisticians. Any information that is stored 
electronically will be kept locked by password access. 


How to contact us to find out more about the study 
If you would like to find out more about the study (even if you decide not to take part), please contact 
Dr Sarah Miller, at the address or phone number below: 


Dr Sarah Miller 

Centre for Effective Education 
School of Education 

Queen's University Belfast 
69-71 University Street 
Belfast 

BT7 1HL 


Telephone: 028 9097 5944 
E-mail: s.j.miller@qub.ac.uk 


Physically Active Lessons Evaluation 
Parent Consent Form 


Please return this slip to the class teacher if you DO NOT want your child to take part in one, some or 
all parts this study. 


Your Name: 


Your Child’s Name: 
Primary School Name: 


Please tick one of the three boxes below to indicate which part(s) of the study you do not wish your 
child to take part in. 


1. If you do not want your child to take take part in any part of the study (including both the focus 
group and the tests of concentration, literacy and numeracy) please tick the box below: 
[4] | do not give my consent for my child to take part in any part of the research 


2. If you do not want your child to take part in the focus group, please tick the box below: 
[14 | do not give my consent for my child to take part in the focus group 


3. If you do not want your child to take part in the tests of concentration, literacy or numeracy, 
please tick the box below: 
Ef) | do not give my consent for my child to take part in the tests of concentration, 
literacy and numeracy tests 


Signed Date 
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Appendix 3: Weekly numeracy plan 


WEEKI.Y NUMERACY PLAN 


Week beginning 23.4.14 


— 


Wednesday 


| Mental and oral storter 


Find ways of meking a rectangle 
with on ares of 36 (on whiteboards 
~ colour squares (LAs) or cubes or 
use table knowledge (MA/HA) 
PAL - Jump the number of sides of 
a shape - name a shope - the 
cnilaren call out the number of 
| ___ sides then jump them, — 


Thursday 


Mental end oral sterter 
Mental maths test Summer 1 
PAL - times tables facts - in pairs 
-ask a question - ¢.9. what x 5= 


15? 
Or unision facts - 30-5 =? 


| 


Friday ee 


| Mental and oral starter 
Investigation - if we know 4 x 10 = 
40, what else do we know? Eg. 4 x 
100 = 400, 0.4 10 =4 etc. Ask 
for division facts too. 


PAL - true or false statements 

| using factors or facts from the 

| investigation. Tog en the spot, star 
jump if correct, spin round +f false, 
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WALT investigate general statements about numbers 


Class = 4P/4b 


WALT extend number sequerices 


WILF count on or back in equal steps to continue the sequence, recegnise and | 


write the rule 


Blue challenge - 44 bk 2 pg 3 (Rapid maths group) 
continue the sequences Ext bottom of the page. 
Sreen challenge - PCM 4 pg 14 extension pg 15 
(double sided sheet - write on the sheet) LP/RD 
Red challenge - PCM, 5 pg 31, copied with PCAM4 pg 15 
(RS in 4D) 

Purple challenge - Number sequences sheet (decimals 
and negative numbers) 


WALT investigating multiples — 
WILF find the factors of given numbers 
Whole class, group and /or ind, work 


Blue Challenge - Use a hundred square to find 
common multiples of 2,5 10 and 3 {colour/circle ete 
on shee? to show) Record the common multiples in 
books LP/RD | 
Green Challenge - A5 pq 46 no.s 1-9 hist common 
multiples using numbers given. Extensien bottom of 
the page. 

Red Challenge - A5 pg 44 - list common multiples 
Extension Who am I? RS in 4P 


Plenary 


Look at the sequence 
on The whiteboord, 
whot is the rule? 

Can you complete the 
Sequence? 

Green pen comment - 
write a target for 
counting in sequences, 


| Plenary 


Factor bugs - ona 
whiteboard, draw the 
factor bugs for 


16 

42 

Or 56 

Chn to fill in bugs on 
the board, showing 


how they worked them 
out 


roel 


WILF say if it's true or false by finding 3 examples to satisfy each 


Whole class, group and or ind, Work 


3 differentiated sets of statements from NNS 
CH 1 page 45 

CH 2 pg 80 

Ch 3 pg 82 


Plenary | 


Choose one of the 
statements thot you 
aren't sure about and 
discuss it, explain your 
reasoning fer your 
answer. 
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Appendix 4: Full results 


Table 7: Raw post-test means for each outcome measure 
Observations (N) Intervention Control 
Outcome Mean (SD) Mean (SD) 


Progress in Maths 228 104.2 (15.7) 104.1 (15.1) 69, 141 


Progress in 213 105.9 (15.2) 107.0 (13.6) 69, 141 
English 


Attitude towards 94 3.07 (0.48) 3.16 (.47) 2,4 
school (Pell and 

Jarvis) 

InCAS Maths 123 104.2 (18.9) 106.7 (18.4) 64, 145 
InCAS English 126 103.8 (13.2) 101.9 (12.4) 70, 131 
INCAS Attitude 88 63.0 (28.1) - -21.9, 100 


Table 8: Models fitted for the literacy variable using clustered robust standard errors 
Parameter estimates (with standard errors) 


Dependent Variable: Literacy Main Model 

Observations 339 339 

Group -0.014 (0.156) 0.198 (0.227) 
Gender -0.093 (0.049) 0.017 (0.040) 
FSM eligibility -0.042 (0.077) -0.039 (0.077) 
Group*Gender -0.177 (0.061) 
Constant 0.005 (0.093) -0.112 (0.121) 
Adjusted R 0.01 0.02 
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Table 9: Models fitted for the numeracy variable using clustered robust standard errors 
Parameter estimates (with standard errors) 


DY =yoX=Vare(=Vaig WEVar le) (-} IW Eta leveled 


Numeracy 


Observations 351 
Group -0.041 (.165) 


Gender 0.015 (.045) 


Gender 


351 
0.112 (0.199) 


0.094 (.024) 


FSM eligibility -0.032 (0.084) -0.029 (0.083) 
Group*Gender 0.132 (0.061) 
Constant 0.022 (0.127) -0.063 (0.136) 
Adjusted R 0.002 0.007 


Table 10: Models fitted for the ‘attitude to school’ variable using clustered robust standard 


_ errors _ 
Parameter estimates (with standard errors) 

Dependent Variable: Attitude WET aMVlefel= Gender 
to School 

| Observations 182 182 
Group -0.155 (0.109) -0.264 (0.158) 
Gender -0.142 (0.043) -0.222 (0.028) 
FSM eligibility 0.009 (0.040) 0.007 (0.040) 
Group*Gender 0.095 (0.053) 
Constant 0.110 (0.095) 0.168 (0.119) 
Adjusted R 0.02 0.03 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
or email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do not 
necessarily reflect those of the Department for Education. 


This document is available for download at www.educationendowmentfoundation.org.uk 


A Education 


Endowment 
Foundation 


The Education Endowment Foundation 
9th Floor, Millbank Tower 

21-24 Millbank 

London 

SW1P 4QP 


www.educationendowmentfoundation.org.uk 


